{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f1302eb5",
   "metadata": {
    "id": "sb_auto_header",
    "tags": [
     "sb_auto_header"
    ]
   },
   "source": [
    "<!-- This cell is automatically updated by tools/tutorial-cell-updater.py -->\n",
    "<!-- The contents are initialized from tutorials/notebook-header.md -->\n",
    "\n",
    "[<img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>](https://colab.research.google.com/github/speechbrain/speechbrain/blob/develop/docs/tutorials/nn/neural-network-adapters.ipynb)\n",
    "to execute or view/download this notebook on\n",
    "[GitHub](https://github.com/speechbrain/speechbrain/tree/develop/docs/tutorials/nn/neural-network-adapters.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f82d21c0-4bdb-4caf-af73-30d0f43ecbab",
   "metadata": {},
   "source": [
    "# Neural Network Adapters for faster low-memory fine-tuning\n",
    "\n",
    "This tutorial covers the SpeechBrain implementation of adapters such as LoRA. This includes how to integrate either SpeechBrain implemented adapters, custom adapters, and adapters from libraries such as PEFT into a pre-trained model.\n",
    "\n",
    "## Prerequisites\n",
    "- [Speech Recognition From Scratch](https://speechbrain.readthedocs.io/en/latest/tutorials/tasks/speech-recognition-from-scratch.html)\n",
    "- A CUDA-enabled device for running this tutorial\n",
    "\n",
    "## Introduction and Background\n",
    "\n",
    "As pre-trained models become larger and more capable, there is growing interest in methods for adapting them for specific tasks in a memory-efficient way, within a reasonable time span. One such technique is freezing the original parameters and inserting a small number of additional parameters into the original model, which are called \"adapters.\" These adapters can often match the performance of full fine-tuning at a fraction of the parameter count, meaning faster and more memory-efficient fine-tuning [1]. One popular technique for doing this is known as Low-Rank Adaptation (LoRA) [2].\n",
    "\n",
    "On the software side, HuggingFace has produced a popular library for adapters called PEFT [3]. Our implementation includes some of the features of this library, as well as including the ability to integrate PEFT adapters into a SpeechBrain model. We'll start with a basic example YAML so you can just try it yourself if you learn better from experimentation.\n",
    "\n",
    "### Relevant bibliography\n",
    "1. N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, \"Parameter-efficient transfer learning for NLP.\" In *International Conference on Machine Learning*, 2019.\n",
    "2. E.J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, \"LoRA: Low-rank adaptation of large language models.\" In *International Conference on Learning Representations*, 2021.\n",
    "3. S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, S. Paul, and  B. Bossan, \"PEFT: State-of-the-art parameter-efficient fine-tuning methods.\" *GitHub Repository*, 2022.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e863ce00-65fa-43eb-9678-cdb2e2c9a219",
   "metadata": {},
   "source": [
    "## Too Long ; Didn't Read\n",
    "\n",
    "The TL;DR of this tutorial is that you should use a section like this in your HyperPyYAML file to create a model with adapters:\n",
    "\n",
    "```yaml\n",
    "adapted_model: !new:speechbrain.nnet.adapters.AdaptedModel\n",
    "    model_to_adapt: !ref <model>\n",
    "    adapter_class: !name:speechbrain.nnet.adapters.LoRA\n",
    "    all_linear: True\n",
    "    unfrozen_layers: [\"conv_1d_*\"]\n",
    "    adapter_kwargs:\n",
    "        rank: 8\n",
    "```\n",
    "\n",
    "Adding this section to the YAML takes the already-defined key `model` and adds a LoRA adapter to every linear layer with the keyword argument `rank=8`.\n",
    "The `all_linear` and `all_conv` arguments simply add adapters to all linear or all convolution layers respectively.\n",
    "By default, this class freezes the parameters of all layers that are not adapted, but the `unfrozen_layers` argument can be used to specify the names\n",
    "of layers that should also be trained, at the cost of a higher parameter count. One can specify specific layers that should be adapted with\n",
    "the `target_layers` argument. These arguments both support unix-style wildcards through the use of python's `fnmatch` library.\n",
    "\n",
    "If the TL;DR isn't enough and you need to work through an example in more detail, continue to the next section."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ca49a1d-5c1b-40db-b7e1-dcc8a85bcce3",
   "metadata": {},
   "source": [
    "## Detailed Tutorial\n",
    "\n",
    "We'll demonstrate how to use adapters on a template recipe, which includes everything necessary for full training. The first step is to pretrain a model so we can later add adapters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a45871e4-e087-486b-af01-411603e43070",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fatal: destination path 'speechbrain' already exists and is not an empty directory.\n",
      "/home/pplantinga/Documents/Repositories/uvenv/bin/python: No module named pip\n"
     ]
    }
   ],
   "source": [
    "!git clone --depth 1 --branch v1.0.2 https://github.com/speechbrain/speechbrain.git\n",
    "!python -m pip install -e speechbrain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23dc826e-49a3-4463-a49d-9c5981849cff",
   "metadata": {},
   "outputs": [],
   "source": [
    "# In order to use speechbrain in this repo we have to add it to the path\n",
    "import os, sys\n",
    "\n",
    "sys.path.append(os.path.join(os.getcwd(), 'speechbrain'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0d88fe51-301b-46d5-858e-fd47877e92ae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/pplantinga/Documents/Repositories/speechbrain/docs/tutorials/nn/speechbrain/templates/speech_recognition/ASR\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/pplantinga/Documents/Repositories/uvenv/lib/python3.12/site-packages/IPython/core/magics/osm.py:417: UserWarning: This is now an optional IPython functionality, setting dhist requires you to install the `pickleshare` library.\n",
      "  self.shell.db['dhist'] = compress_dhist(dhist)[-100:]\n"
     ]
    }
   ],
   "source": [
    "%cd speechbrain/templates/speech_recognition/ASR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9dabda46-19ae-41b9-ab0f-6b9995b1e27f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:speechbrain.utils.seed:Setting seed to 2602\n",
      "WARNING:speechbrain.utils.train_logger:torchvision is not available - cannot save figures\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/utils/autocast.py:68: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.\n",
      "  wrapped_fwd = torch.cuda.amp.custom_fwd(fwd, cast_inputs=cast_inputs)\n",
      "speechbrain.core - Beginning experiment!\n",
      "speechbrain.core - Experiment folder: results/CRDNN_BPE_960h_LM/2602\n",
      "mini_librispeech_prepare - Preparation completed in previous run, skipping.\n",
      "../data/noise/data.zip exists. Skipping download\n",
      "../data/rir/data.zip exists. Skipping download\n",
      "speechbrain.utils.fetching - Fetch lm.ckpt: Fetching from HuggingFace Hub 'speechbrain/asr-crdnn-rnnlm-librispeech' if not cached\n",
      "speechbrain.utils.fetching - Fetch lm.ckpt: Fetching from HuggingFace Hub 'speechbrain/asr-crdnn-rnnlm-librispeech' if not cached\n",
      "speechbrain.utils.fetching - Fetch tokenizer.ckpt: Fetching from HuggingFace Hub 'speechbrain/asr-crdnn-rnnlm-librispeech' if not cached\n",
      "speechbrain.utils.fetching - Fetch tokenizer.ckpt: Fetching from HuggingFace Hub 'speechbrain/asr-crdnn-rnnlm-librispeech' if not cached\n",
      "speechbrain.utils.fetching - Fetch asr.ckpt: Fetching from HuggingFace Hub 'speechbrain/asr-crdnn-rnnlm-librispeech' if not cached\n",
      "speechbrain.utils.fetching - Fetch asr.ckpt: Fetching from HuggingFace Hub 'speechbrain/asr-crdnn-rnnlm-librispeech' if not cached\n",
      "speechbrain.utils.parameter_transfer - Loading pretrained files for: lm, tokenizer, model\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/utils/checkpoints.py:199: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  state_dict = torch.load(path, map_location=device)\n",
      "speechbrain.core - Info: ckpt_interval_minutes arg from hparam file is used\n",
      "speechbrain.core - Gradscaler enabled: False. Using precision: fp32.\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/core.py:793: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.\n",
      "  self.scaler = torch.cuda.amp.GradScaler(enabled=gradscaler_enabled)\n",
      "speechbrain.core - ASR Model Statistics:\n",
      "* Total Number of Trainable Parameters: 173.0M\n",
      "* Total Number of Parameters: 173.0M\n",
      "* Trainable Parameters represent 100.0000% of the total size.\n",
      "speechbrain.utils.checkpoints - Would load a checkpoint here, but none found yet.\n",
      "speechbrain.utils.epoch_loop - Going into epoch 1\n",
      "speechbrain.augment.augmenter - No augmentation is applied because the augmentation start index is greater than or equal to the number of examples in the input batch.\n",
      "100%|████████████████████████| 760/760 [08:28<00:00,  1.49it/s, train_loss=1.35]\n",
      "100%|█████████████████████████████████████████| 545/545 [01:28<00:00,  6.18it/s]\n",
      "speechbrain.utils.train_logger - epoch: 1, lr: 1.00e+00 - train loss: 1.35 - valid loss: 1.31, valid CER: 7.71, valid WER: 20.06\n",
      "speechbrain.utils.checkpoints - Saved an end-of-epoch checkpoint in results/CRDNN_BPE_960h_LM/2602/save/CKPT+2024-10-08+11-08-06+00\n",
      "speechbrain.utils.checkpoints - Loading a checkpoint from results/CRDNN_BPE_960h_LM/2602/save/CKPT+2024-10-08+11-08-06+00\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/utils/checkpoints.py:199: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  state_dict = torch.load(path, map_location=device)\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/nnet/schedulers.py:240: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  data = torch.load(path)\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/processing/features.py:1311: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  stats = torch.load(path, map_location=device)\n",
      "100%|███████████████████████████████████████| 1310/1310 [09:25<00:00,  2.32it/s]\n",
      "speechbrain.utils.train_logger - Epoch loaded: 1 - test loss: 1.30, test CER: 5.75, test WER: 17.57\n",
      "speechbrain.utils.checkpoints - Saved an end-of-epoch checkpoint in results/CRDNN_BPE_960h_LM/2602/save/CKPT+latest\n"
     ]
    }
   ],
   "source": [
    "!python train.py train.yaml --number_of_epochs=1 --batch_size=2 --test_scorer \"!ref <valid_scorer>\" --enable_add_reverb=False --enable_add_noise=False #To speed up"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9782e5b-5498-487f-8055-0b73f75a8ec8",
   "metadata": {},
   "source": [
    "## Inference\n",
    "\n",
    "To prove that this is working, let's just perform inference on one file. This code taken from `transcribe_file.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9dd9de34-e7d2-4fdd-beb7-d7ab07551c04",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:speechbrain.utils.fetching:Fetch lm.ckpt: Fetching from HuggingFace Hub 'speechbrain/asr-crdnn-rnnlm-librispeech' if not cached\n",
      "INFO:speechbrain.utils.fetching:Fetch tokenizer.ckpt: Fetching from HuggingFace Hub 'speechbrain/asr-crdnn-rnnlm-librispeech' if not cached\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/utils/autocast.py:68: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.\n",
      "  wrapped_fwd = torch.cuda.amp.custom_fwd(fwd, cast_inputs=cast_inputs)\n",
      "INFO:speechbrain.utils.parameter_transfer:Loading pretrained files for: lm, tokenizer, model, normalizer\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/utils/checkpoints.py:199: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  state_dict = torch.load(path, map_location=device)\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/processing/features.py:1311: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  stats = torch.load(path, map_location=device)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'THE METAL FOREST IS IN THE GREAT DOMED CAVERN THE LARGEST IN ALL OUR DOMINIONS REPLIED CALICO ⁇ '"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from speechbrain.inference.ASR import EncoderDecoderASR\n",
    "from speechbrain.utils.fetching import fetch, LocalStrategy\n",
    "\n",
    "# Ensure all the needed files end up in the same place to load with the transcriber\n",
    "save_dir = os.path.abspath(\"results/CRDNN_BPE_960h_LM/2602/save/CKPT+latest\")\n",
    "fetch(\"lm.ckpt\", \"speechbrain/asr-crdnn-rnnlm-librispeech\", save_dir, local_strategy=LocalStrategy.SYMLINK)\n",
    "fetch(\"tokenizer.ckpt\", \"speechbrain/asr-crdnn-rnnlm-librispeech\", save_dir, local_strategy=LocalStrategy.SYMLINK)\n",
    "fetch(\"inference.yaml\", os.getcwd(), save_dir, local_strategy=LocalStrategy.SYMLINK)\n",
    "\n",
    "transcriber = EncoderDecoderASR.from_hparams(source=save_dir, hparams_file=\"inference.yaml\")\n",
    "speech_file = \"../data/LibriSpeech/dev-clean-2/1272/135031/1272-135031-0015.flac\"\n",
    "transcriber.transcribe_file(speech_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "179c13b9-5a32-412e-a099-951163f4ed3c",
   "metadata": {},
   "source": [
    "## Adding adapters\n",
    "\n",
    "So now that we've proved that the model is at least working, let's go ahead and add adapters. We basically need to create a new yaml file adding adapters to the model and then train with this new yaml file. To do this we'll just load the old yaml file and then we'll change all the parts necessary to train the adapted model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b0904742-9f03-46c4-88bb-fd7cb4326686",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting train_lora.patch\n"
     ]
    }
   ],
   "source": [
    "%%writefile train_lora.patch\n",
    "--- train.yaml\t2024-10-07 19:23:49.839501714 -0400\n",
    "+++ train_lora.yaml\t2024-10-07 19:25:40.340933091 -0400\n",
    "@@ -30,7 +30,7 @@\n",
    " NOISE_DATASET_URL: https://www.dropbox.com/scl/fi/a09pj97s5ifan81dqhi4n/noises.zip?rlkey=j8b0n9kdjdr32o1f06t0cw5b7&dl=1\n",
    " RIR_DATASET_URL: https://www.dropbox.com/scl/fi/linhy77c36mu10965a836/RIRs.zip?rlkey=pg9cu8vrpn2u173vhiqyu743u&dl=1\n",
    " \n",
    "-output_folder: !ref results/CRDNN_BPE_960h_LM/<seed>\n",
    "+output_folder: !ref results/crdnn_lora/<seed>\n",
    " test_wer_file: !ref <output_folder>/wer_test.txt\n",
    " save_folder: !ref <output_folder>/save\n",
    " train_log: !ref <output_folder>/train_log.txt\n",
    "@@ -41,7 +41,7 @@\n",
    " # speechbrain HuggingFace repository. However, a local path pointing to a\n",
    " # directory containing the lm.ckpt and tokenizer.ckpt may also be specified\n",
    " # instead. E.g if you want to use your own LM / tokenizer.\n",
    "-pretrained_path: speechbrain/asr-crdnn-rnnlm-librispeech\n",
    "+pretrained_path: results/CRDNN_BPE_960h_LM/2602/save/CKPT+latest\n",
    " \n",
    " \n",
    " # Path where data manifest files will be stored. The data manifest files are created by the\n",
    "@@ -481,10 +481,9 @@\n",
    "     ctc_lin: !ref <ctc_lin>\n",
    "     seq_lin: !ref <seq_lin>\n",
    "     normalize: !ref <normalize>\n",
    "-    lm_model: !ref <lm_model>\n",
    " \n",
    " # Gathering all the submodels in a single model object.\n",
    "-model: !new:torch.nn.ModuleList\n",
    "+model_pretrained: !new:torch.nn.ModuleList\n",
    "     - - !ref <encoder>\n",
    "       - !ref <embedding>\n",
    "       - !ref <decoder>\n",
    "@@ -629,8 +628,31 @@\n",
    "     loadables:\n",
    "         lm: !ref <lm_model>\n",
    "         tokenizer: !ref <tokenizer>\n",
    "-        model: !ref <model>\n",
    "+        model: !ref <model_pretrained>\n",
    "     paths:\n",
    "         lm: !ref <pretrained_path>/lm.ckpt\n",
    "         tokenizer: !ref <pretrained_path>/tokenizer.ckpt\n",
    "-        model: !ref <pretrained_path>/asr.ckpt\n",
    "+        model: !ref <pretrained_path>/model.ckpt\n",
    "+\n",
    "+new_encoder: !new:speechbrain.nnet.adapters.AdaptedModel\n",
    "+    model_to_adapt: !ref <encoder>\n",
    "+    adapter_class: !name:speechbrain.nnet.adapters.LoRA\n",
    "+    all_linear: True\n",
    "+    manual_adapter_insertion: True\n",
    "+    adapter_kwargs:\n",
    "+        rank: 8\n",
    "+\n",
    "+new_decoder: !new:speechbrain.nnet.adapters.AdaptedModel\n",
    "+    model_to_adapt: !ref <decoder>\n",
    "+    adapter_class: !name:speechbrain.nnet.adapters.LoRA\n",
    "+    all_linear: True\n",
    "+    manual_adapter_insertion: True\n",
    "+    adapter_kwargs:\n",
    "+        rank: 8\n",
    "+\n",
    "+model: !new:torch.nn.ModuleList\n",
    "+    - - !ref <new_encoder>\n",
    "+      - !ref <embedding>\n",
    "+      - !ref <new_decoder>\n",
    "+      - !ref <ctc_lin>\n",
    "+      - !ref <seq_lin>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8517a99a-1dd0-46e9-84b6-2e503e2aa270",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "patching file train_lora.yaml (read from train.yaml)\n"
     ]
    }
   ],
   "source": [
    "!patch train.yaml -i train_lora.patch -o train_lora.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3645cf7-a902-46dd-b4ea-b5a60de556d8",
   "metadata": {},
   "source": [
    "Because we are loading the pretrained parameters using the pretrainer, we have to insert this code to insert the adapters after the pretrained parameters have been loaded.\n",
    "\n",
    "This is the reason for the `manual_adapter_insertion: True` in the yaml and the following brief change to the training code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "36508f13-7423-47eb-98dd-40274f1990b6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting train_lora_py.patch\n"
     ]
    }
   ],
   "source": [
    "%%writefile train_lora_py.patch\n",
    "--- train.py\t2024-10-07 14:57:21.534381751 -0400\n",
    "+++ train_lora.py\t2024-10-07 19:33:12.839895913 -0400\n",
    "@@ -473,6 +473,8 @@\n",
    "     # the path given in the YAML file). The tokenizer is loaded at the same time.\n",
    "     hparams[\"pretrainer\"].collect_files()\n",
    "     hparams[\"pretrainer\"].load_collected()\n",
    "+    hparams[\"new_encoder\"].insert_adapters()\n",
    "+    hparams[\"new_decoder\"].insert_adapters()\n",
    " \n",
    "     # Trainer initialization\n",
    "     asr_brain = ASR("
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6b1b9c73-36a1-4212-9481-654f97b5f441",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "patching file train_lora.py (read from train.py)\n"
     ]
    }
   ],
   "source": [
    "!patch train.py -i train_lora_py.patch -o train_lora.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8792396c-c05c-4589-b3f3-149cfd98fb34",
   "metadata": {},
   "source": [
    "## Training the adapted model\n",
    "\n",
    "Training works identically to before, using the updated lora file. The adapted model is designed to work as an in-place replacement. Notice how the number of trainable parameters is reduced to close to 1% of the original parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d088fc19-5f0b-4160-b470-0a4198f0f0e8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:speechbrain.utils.seed:Setting seed to 2602\n",
      "WARNING:speechbrain.utils.train_logger:torchvision is not available - cannot save figures\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/utils/autocast.py:68: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.\n",
      "  wrapped_fwd = torch.cuda.amp.custom_fwd(fwd, cast_inputs=cast_inputs)\n",
      "speechbrain.core - Beginning experiment!\n",
      "speechbrain.core - Experiment folder: results/crdnn_lora/2602\n",
      "mini_librispeech_prepare - Preparation completed in previous run, skipping.\n",
      "../data/noise/data.zip exists. Skipping download\n",
      "../data/rir/data.zip exists. Skipping download\n",
      "speechbrain.utils.parameter_transfer - Loading pretrained files for: lm, tokenizer, model\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/utils/checkpoints.py:199: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  state_dict = torch.load(path, map_location=device)\n",
      "speechbrain.core - Info: ckpt_interval_minutes arg from hparam file is used\n",
      "speechbrain.core - Gradscaler enabled: False. Using precision: fp32.\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/core.py:793: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.\n",
      "  self.scaler = torch.cuda.amp.GradScaler(enabled=gradscaler_enabled)\n",
      "speechbrain.core - ASR Model Statistics:\n",
      "* Total Number of Trainable Parameters: 1.8M\n",
      "* Total Number of Parameters: 120.0M\n",
      "* Trainable Parameters represent 1.4807% of the total size.\n",
      "speechbrain.utils.checkpoints - Would load a checkpoint here, but none found yet.\n",
      "speechbrain.utils.epoch_loop - Going into epoch 1\n",
      "speechbrain.augment.augmenter - No augmentation is applied because the augmentation start index is greater than or equal to the number of examples in the input batch.\n",
      "100%|███████████████████████████| 760/760 [04:09<00:00,  3.04it/s, train_loss=1]\n",
      "100%|█████████████████████████████████████████| 545/545 [01:40<00:00,  5.42it/s]\n",
      "speechbrain.utils.train_logger - epoch: 1, lr: 1.00e+00 - train loss: 1.00 - valid loss: 1.26, valid CER: 7.29, valid WER: 19.16\n",
      "speechbrain.utils.checkpoints - Saved an end-of-epoch checkpoint in results/crdnn_lora/2602/save/CKPT+2024-10-08+11-23-53+00\n",
      "speechbrain.utils.checkpoints - Loading a checkpoint from results/crdnn_lora/2602/save/CKPT+2024-10-08+11-23-53+00\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/utils/checkpoints.py:199: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  state_dict = torch.load(path, map_location=device)\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/nnet/schedulers.py:240: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  data = torch.load(path)\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/processing/features.py:1311: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  stats = torch.load(path, map_location=device)\n",
      "100%|███████████████████████████████████████| 1310/1310 [12:55<00:00,  1.69it/s]\n",
      "speechbrain.utils.train_logger - Epoch loaded: 1 - test loss: 1.26, test CER: 5.62, test WER: 17.05\n",
      "speechbrain.utils.checkpoints - Saved an end-of-epoch checkpoint in results/crdnn_lora/2602/save/CKPT+latest\n"
     ]
    }
   ],
   "source": [
    "!python train_lora.py train_lora.yaml --number_of_epochs=1 --batch_size=2 --test_scorer \"!ref <valid_scorer>\" --enable_add_reverb=False --enable_add_noise=False #To speed up"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f960be7a-6edb-47c1-bd8d-c7c0d486c81d",
   "metadata": {},
   "source": [
    "## Custom adapter\n",
    "\n",
    "We designed this so that you could replace the SpeechBrain adapter with a `peft` adapter:\n",
    "\n",
    "```diff\n",
    "new_encoder: !new:speechbrain.nnet.adapters.AdaptedModel\n",
    "    model_to_adapt: !ref <encoder>\n",
    "-   adapter_class: !name:speechbrain.nnet.adapters.LoRA\n",
    "+   adapter_class: !name:peft.tuners.lora.layer.Linear\n",
    "    manual_adapter_insertion: True\n",
    "    adapter_kwargs:\n",
    "-       rank: 16\n",
    "+       r: 16\n",
    "+       adapter_name: lora\n",
    "```\n",
    "\n",
    "But this trains exactly the same thing as before, so no need for us to go through the whole thing. Perhaps more interesting is designing a custom adapter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f9682f70-489a-4a1d-b8c6-1c73d98a824d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting pool_lora.py\n"
     ]
    }
   ],
   "source": [
    "%%file pool_lora.py\n",
    "\n",
    "import torch\n",
    "\n",
    "class PoolLoRA(torch.nn.Module):\n",
    "    def __init__(self, target_module, stride=2, rank=16, alpha=1.0):\n",
    "        super().__init__()\n",
    "\n",
    "        input_size = target_module.weight.data.shape[1]\n",
    "        output_size = target_module.weight.data.shape[0]\n",
    "        \n",
    "        # Disable gradient for pretrained module\n",
    "        self.pretrained_module = target_module\n",
    "        for param in self.pretrained_module.parameters():\n",
    "            param.requires_grad = False\n",
    "        device = target_module.weight.device\n",
    "\n",
    "        self.adapter_down_scale = torch.nn.AvgPool1d(kernel_size=stride)\n",
    "        self.adapter_down_proj = torch.nn.Linear(\n",
    "            input_size // stride, rank, bias=False, device=device\n",
    "        )   \n",
    "        self.adapter_up_proj = torch.nn.Linear(\n",
    "            rank, output_size, bias=False, device=device\n",
    "        )   \n",
    "        self.adapter_up_proj.weight.data.fill_(0.0)\n",
    "\n",
    "        self.scaling = alpha / rank\n",
    "\n",
    "    def forward(self, x: torch.Tensor):\n",
    "        \"\"\"Applies the LoRA Adapter.\n",
    "\n",
    "        Arguments\n",
    "        ---------\n",
    "        x: torch.Tensor\n",
    "            Input tensor to the adapter module.\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        The linear outputs\n",
    "        \"\"\"\n",
    "        x_pretrained = self.pretrained_module(x)\n",
    "\n",
    "        x_downsample = self.adapter_down_proj(self.adapter_down_scale(x))\n",
    "        x_pool_lora = self.adapter_up_proj(x_downsample)\n",
    "        \n",
    "        return x_pretrained + x_pool_lora * self.scaling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c2e702a9-c07d-4a76-94bc-847b8f890579",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting train_pool_lora.patch\n"
     ]
    }
   ],
   "source": [
    "%%writefile train_pool_lora.patch\n",
    "--- train_lora.yaml\t2024-10-07 22:44:02.767830301 -0400\n",
    "+++ train_pool_lora.yaml\t2024-10-07 22:41:30.602641301 -0400\n",
    "@@ -30,7 +30,7 @@\n",
    " NOISE_DATASET_URL: https://www.dropbox.com/scl/fi/a09pj97s5ifan81dqhi4n/noises.zip?rlkey=j8b0n9kdjdr32o1f06t0cw5b7&dl=1\n",
    " RIR_DATASET_URL: https://www.dropbox.com/scl/fi/linhy77c36mu10965a836/RIRs.zip?rlkey=pg9cu8vrpn2u173vhiqyu743u&dl=1\n",
    " \n",
    "-output_folder: !ref results/crdnn_lora/<seed>\n",
    "+output_folder: !ref results/crdnn_pool_lora/<seed>\n",
    " test_wer_file: !ref <output_folder>/wer_test.txt\n",
    " save_folder: !ref <output_folder>/save\n",
    " train_log: !ref <output_folder>/train_log.txt\n",
    "@@ -636,19 +636,21 @@\n",
    " \n",
    " new_encoder: !new:speechbrain.nnet.adapters.AdaptedModel\n",
    "     model_to_adapt: !ref <encoder>\n",
    "-    adapter_class: !name:speechbrain.nnet.adapters.LoRA\n",
    "+    adapter_class: !name:pool_lora.PoolLoRA\n",
    "     all_linear: True\n",
    "     manual_adapter_insertion: True\n",
    "     adapter_kwargs:\n",
    "-        rank: 8\n",
    "+        stride: 2\n",
    "+        rank: 16\n",
    " \n",
    " new_decoder: !new:speechbrain.nnet.adapters.AdaptedModel\n",
    "     model_to_adapt: !ref <decoder>\n",
    "-    adapter_class: !name:speechbrain.nnet.adapters.LoRA\n",
    "+    adapter_class: !name:pool_lora.PoolLoRA\n",
    "     all_linear: True\n",
    "     manual_adapter_insertion: True\n",
    "     adapter_kwargs:\n",
    "-        rank: 8\n",
    "+        stride: 2\n",
    "+        rank: 16\n",
    " \n",
    " model: !new:torch.nn.ModuleList\n",
    "     - - !ref <new_encoder>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "532bfcda-4a30-463d-8cfe-eed96d28732c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "patching file train_pool_lora.yaml (read from train_lora.yaml)\n"
     ]
    }
   ],
   "source": [
    "!patch train_lora.yaml -i train_pool_lora.patch -o train_pool_lora.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "56aefd64-1325-4891-a9a4-1c4e85691b96",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:speechbrain.utils.seed:Setting seed to 2602\n",
      "WARNING:speechbrain.utils.train_logger:torchvision is not available - cannot save figures\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/utils/autocast.py:68: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.\n",
      "  wrapped_fwd = torch.cuda.amp.custom_fwd(fwd, cast_inputs=cast_inputs)\n",
      "speechbrain.core - Beginning experiment!\n",
      "speechbrain.core - Experiment folder: results/crdnn_pool_lora/2602\n",
      "mini_librispeech_prepare - Preparation completed in previous run, skipping.\n",
      "../data/noise/data.zip exists. Skipping download\n",
      "../data/rir/data.zip exists. Skipping download\n",
      "speechbrain.utils.parameter_transfer - Loading pretrained files for: lm, tokenizer, model\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/utils/checkpoints.py:199: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  state_dict = torch.load(path, map_location=device)\n",
      "speechbrain.core - Info: ckpt_interval_minutes arg from hparam file is used\n",
      "speechbrain.core - Gradscaler enabled: False. Using precision: fp32.\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/core.py:793: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.\n",
      "  self.scaler = torch.cuda.amp.GradScaler(enabled=gradscaler_enabled)\n",
      "speechbrain.core - ASR Model Statistics:\n",
      "* Total Number of Trainable Parameters: 1.8M\n",
      "* Total Number of Parameters: 120.0M\n",
      "* Trainable Parameters represent 1.5210% of the total size.\n",
      "speechbrain.utils.checkpoints - Would load a checkpoint here, but none found yet.\n",
      "speechbrain.utils.epoch_loop - Going into epoch 1\n",
      "speechbrain.augment.augmenter - No augmentation is applied because the augmentation start index is greater than or equal to the number of examples in the input batch.\n",
      "100%|████████████████████████| 760/760 [04:19<00:00,  2.93it/s, train_loss=0.98]\n",
      "100%|█████████████████████████████████████████| 545/545 [01:44<00:00,  5.24it/s]\n",
      "speechbrain.utils.train_logger - epoch: 1, lr: 1.00e+00 - train loss: 9.80e-01 - valid loss: 1.26, valid CER: 7.18, valid WER: 18.92\n",
      "speechbrain.utils.checkpoints - Saved an end-of-epoch checkpoint in results/crdnn_pool_lora/2602/save/CKPT+2024-10-08+11-43-00+00\n",
      "speechbrain.utils.checkpoints - Loading a checkpoint from results/crdnn_pool_lora/2602/save/CKPT+2024-10-08+11-43-00+00\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/utils/checkpoints.py:199: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  state_dict = torch.load(path, map_location=device)\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/nnet/schedulers.py:240: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  data = torch.load(path)\n",
      "/home/pplantinga/Documents/Repositories/speechbrain/speechbrain/processing/features.py:1311: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  stats = torch.load(path, map_location=device)\n",
      "100%|███████████████████████████████████████| 1310/1310 [14:07<00:00,  1.55it/s]\n",
      "speechbrain.utils.train_logger - Epoch loaded: 1 - test loss: 1.25, test CER: 5.61, test WER: 16.99\n",
      "speechbrain.utils.checkpoints - Saved an end-of-epoch checkpoint in results/crdnn_pool_lora/2602/save/CKPT+latest\n"
     ]
    }
   ],
   "source": [
    "!python train_lora.py train_pool_lora.yaml --number_of_epochs=1 --batch_size=2 --test_scorer \"!ref <valid_scorer>\" --enable_add_reverb=False --enable_add_noise=False #To speed up"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21ef247c-3022-4b65-8cd0-86d01a618b79",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "That's it, thanks for following along! Go forth and make cool adapters."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11474af6",
   "metadata": {
    "id": "sb_auto_footer",
    "tags": [
     "sb_auto_footer"
    ]
   },
   "source": [
    "## Citing SpeechBrain\n",
    "\n",
    "If you use SpeechBrain in your research or business, please cite it using the following BibTeX entry:\n",
    "\n",
    "```bibtex\n",
    "@misc{speechbrainV1,\n",
    "  title={Open-Source Conversational AI with {SpeechBrain} 1.0},\n",
    "  author={Mirco Ravanelli and Titouan Parcollet and Adel Moumen and Sylvain de Langen and Cem Subakan and Peter Plantinga and Yingzhi Wang and Pooneh Mousavi and Luca Della Libera and Artem Ploujnikov and Francesco Paissan and Davide Borra and Salah Zaiem and Zeyu Zhao and Shucong Zhang and Georgios Karakasidis and Sung-Lin Yeh and Pierre Champion and Aku Rouhe and Rudolf Braun and Florian Mai and Juan Zuluaga-Gomez and Seyed Mahed Mousavi and Andreas Nautsch and Xuechen Liu and Sangeet Sagar and Jarod Duret and Salima Mdhaffar and Gaelle Laperriere and Mickael Rouvier and Renato De Mori and Yannick Esteve},\n",
    "  year={2024},\n",
    "  eprint={2407.00463},\n",
    "  archivePrefix={arXiv},\n",
    "  primaryClass={cs.LG},\n",
    "  url={https://arxiv.org/abs/2407.00463},\n",
    "}\n",
    "@misc{speechbrain,\n",
    "  title={{SpeechBrain}: A General-Purpose Speech Toolkit},\n",
    "  author={Mirco Ravanelli and Titouan Parcollet and Peter Plantinga and Aku Rouhe and Samuele Cornell and Loren Lugosch and Cem Subakan and Nauman Dawalatabad and Abdelwahab Heba and Jianyuan Zhong and Ju-Chieh Chou and Sung-Lin Yeh and Szu-Wei Fu and Chien-Feng Liao and Elena Rastorgueva and François Grondin and William Aris and Hwidong Na and Yan Gao and Renato De Mori and Yoshua Bengio},\n",
    "  year={2021},\n",
    "  eprint={2106.04624},\n",
    "  archivePrefix={arXiv},\n",
    "  primaryClass={eess.AS},\n",
    "  note={arXiv:2106.04624}\n",
    "}\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
